Image processing device, imaging system provided therewith, and calibration method

ABSTRACT

An image processing device includes a stitching processing unit that composites composition source images generated from captured images under a preset processing condition to generate a composite image, a screen generation unit that generates a screen and outputs the composite image to a display input device, a touch operation determination unit that determines a camera image region to be adjusted based on a detection result of a touch operation on the screen and determines an adjustment item according to an operation mode of the touch operation, and a processing condition setting unit that sets a temporary processing condition related to the camera image region to be adjusted. The stitching processing unit generates the composition source image from the captured image of the camera corresponding to the camera image region to be adjusted under the temporary processing condition and updates the composite image by compositing the composition source images.

This is a continuation of U.S. patent application Ser. No. 16/088,697,filed Sep. 26, 2018, which is a National Stage Entry of InternationalPatent Appl. No. PCT/JP2017/014327, filed Apr. 6, 2017, which claimspriority to Japanese Appl. No. 2016-087443, filed Apr. 25, 2016. Thedisclosure of each of the above-mentioned documents, including thespecification, drawings, and claims, is incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing device thatoutputs a composite image by performing an image composition process onan image for each of a plurality of cameras, an imaging system providedtherewith, and a calibration method of securing consistency betweencamera image regions in the composite image.

BACKGROUND ART

By performing a so-called stitching process of generating one compositeimage by compositing images captured by a plurality of cameras, awide-angle image which cannot be obtained by one camera can begenerated. In the stitching process, two adjacent cameras are arrangedso that respective imaging areas of the cameras partially overlap witheach other and image regions of boundary portions corresponding tooverlapping portions of the imaging areas of the two cameras aresuperimposed or trimming is appropriately performed on the image regionsto composite the images.

On the other hand, when subjects with different distances from thecamera exist, that is, the subject of distant view and the subject ofclose-range view exist in the overlapping portion of the imaging areasof the two cameras, a positional relationship of an image of the subjectof distant view and an image of the subject of close-range viewdeviates, so-called a disparity occurs, the image of the subject ofdistant view duplicately appears, and a defect of losing a part of theimage of the subject of distant view occurs between the respectivecaptured images of the two cameras. Therefore, in the stitching,disparity correction for suppressing the image defect caused by thedisparity is performed.

Regarding the disparity correction, there is known a technique ofobtaining the positional relationship between the images of the subjectswhich appear in the respective captured images of the two cameras byblock matching based on edges and feature amounts and performing thedisparity correction for deforming the image based on the information(see PTL 1). Specifically, in the technique, a stitching point whichdefines a deformation degree of the image during the disparitycorrection is changed for each of frames so as to appropriately generatea composite image for each of the frames.

Meanwhile, in a camera unit in which the plurality of cameras aremounted, when assembling an optical system constituting the camera, anerror occurs in a mechanical disposition of positions, angles, and thelike. This error of the mechanical disposition causes a state in whichthe image regions of the respective cameras are not appropriatelyconsistent in the composite image obtained by the image compositionprocess and causes deterioration of a quality of the composite image.

On the other hand, it is difficult for a user to adjust the mechanicaldisposition itself after product shipment. In addition, the error of themechanical disposition is changed by a difference in an installationenvironment and is also changed as time passes. Therefore, by changing aprocessing condition of the image process, calibration of ensuringconsistency between the camera image regions in the composite image isperformed, and in particular, a technique which allows the user toconveniently perform adjustment work for the calibration at anappropriate timing such as immediately before photographing is desired.

The present disclosure enables the user to conveniently perform theadjustment work for the calibration of ensuring the consistency betweenthe camera image regions in the composite image.

CITATION LIST Patent Literature

PTL 1: Japanese Patent Unexamined Publication No. 2010-50842

SUMMARY OF THE INVENTION

According to the present disclosure, there is provided an imageprocessing device that outputs a composite image by performing an imagecomposition process on an image for each of a plurality of cameras, thedevice including: an image processing unit that generates a plurality ofcomposition source images from each of captured images for each of thecameras based on a preset processing condition and performs the imagecomposition process on the plurality of composition source images togenerate a composite image for preview; a screen generation unit thatgenerates a preview display screen on which the composite image forpreview is displayed and outputs the preview display screen to a displayinput device; a touch operation determination unit that determines acamera image region to be adjusted in the composite image for previewbased on a detection result of a touch operation which a user performson the preview display screen displayed on the display input device anddetermines an adjustment item according to an operation mode of thetouch operation; and a processing condition setting unit that sets atemporary processing condition related to the camera image region to beadjusted based on a determination result of the touch operationdetermination unit, in which the image processing unit generates thecomposition source image from the captured image of the cameracorresponding to the camera image region to be adjusted based on thetemporary processing condition and updates the composite image forpreview by performing the image composition process on the compositionsource image.

In addition, an imaging system according to the present disclosureincludes the image processing device, the plurality of cameras, and thedisplay input device.

In addition, according to the present disclosure, there is provided acalibration method of securing consistency between camera image regionsin a composite image generated by performing an image compositionprocess on an image for each of a plurality of cameras, the methodincluding: generating a plurality of composition source images from eachof captured images for each of the cameras based on a preset processingcondition and performing the image composition process on the pluralityof composition source images to generate a composite image for preview;generating a preview display screen on which the composite image forpreview is displayed and outputting the preview display screen to adisplay input device; determining the camera image region to be adjustedin the composite image for preview based on a detection result of atouch operation which a user performs on the preview display screendisplayed on the display input device and determining an adjustment itemaccording to an operation mode of the touch operation; setting atemporary processing condition related to the camera image region to beadjusted based on a determination result; and generating the compositionsource image from the captured image of the camera corresponding to theimage region to be adjusted based on the temporary processing conditionand updating the composite image for preview by performing the imagecomposition process on the composition source image.

According to the present disclosure, it is possible to executecalibration of ensuring the consistency between the camera image regionsin the composite image by the user performing the touch operation on thepreview display screen displayed on the display input device.Accordingly, it is possible for the user to conveniently performadjustment work for the calibration by an intuitive operation.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an overall configuration diagram of an imaging systemaccording to the present embodiment.

FIG. 2 is an explanatory diagram illustrating a composite imagegenerated by image processing device 3 and displayed on display inputdevice 4.

FIG. 3 is a functional block diagram illustrating a schematicconfiguration of image processing device 3.

FIG. 4 is an explanatory diagram illustrating an outline of a stitchingprocess performed by image processing device 3.

FIG. 5 is an explanatory diagram schematically illustrating states of animage in a case where disparity correction is not executed and a casewhere the disparity correction is executed.

FIGS. 6A-6D illustrate an art of adjusting camera image regions 12.

FIG. 7 is an explanatory diagram illustrating a list of adjustment itemsaccording to an operation mode of a touch operation.

FIGS. 8A-8D are explanatory diagrams illustrating the touch operation ina case of individually adjusting each of camera image regions 12 ofhorizontal cameras 1.

FIG. 9 is an explanatory diagram illustrating the touch operation in acase of individually adjusting camera image regions 12 of upward cameras1.

FIG. 10 is an explanatory diagram illustrating the touch operation in acase of integrally adjusting all of camera image regions 12.

FIGS. 11A-11B are explanatory diagrams illustrating the touch operationin a case of integrally adjusting all of camera image regions 12.

FIG. 12 is a flowchart illustrating a flow of a process executed byimage processing device 3 according to the touch operation of a user.

FIG. 13 is a flowchart illustrating a flow of a process executed byimage processing device 3 according to the touch operation of the user.

FIG. 14 is a flowchart illustrating a flow of a process executed byimage processing device 3 according to the touch operation of the user.

FIG. 15 is an explanatory diagram illustrating an image process ofimproving visibility of camera image region 12 to be adjusted.

FIG. 16 is an explanatory diagram illustrating control of enabling theuser to recognize a just fit state of camera image region 12 to beadjusted.

FIG. 17 is an explanatory diagram illustrating the touch operation whena tone and brightness of camera image region 12 are adjusted.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be describedwith reference to drawings.

FIG. 1 is an overall configuration diagram of an imaging systemaccording to the present embodiment. FIG. 2 is an explanatory diagramillustrating a composite image generated by an image processing device 3and displayed on a display input device 4.

As illustrated in FIG. 1, the present imaging system includes cameraunit 2 having a plurality of cameras 1, image processing device 3, anddisplay input device 4.

Seven cameras 1 of one upward camera 1 which is disposed so that anoptical axis is upward in an approximately vertical direction and sixhorizontal cameras 1 which are arranged at equal intervals in acircumferential direction so that an optical axis is radial in anapproximately horizontal direction are mounted on camera unit 2. Each ofcameras 1 has a wide angle of view (for example, 120°) and is arrangedso that two imaging areas adjacent to each other partially overlap witheach other.

As illustrated in FIG. 2, image processing device 3 performs a stitchingprocess of compositing captured images captured by each of cameras 1 togenerate one composite image 11. Composite image 11 is in a state inwhich camera image regions 12 are combined based on each of capturedimages captured by each of cameras 1. In addition, the stitching processis performed on each of frames, a composite image of each of the framesis output to display input device 4, and the composite images aredisplayed on display input device 4 as a video.

In the present embodiment, image processing device 3 generates thecomposite image in real time and outputs the composite image to displayinput device 4 so that a user can see the composite image in real time,but the composite image generated by image processing device 3 may bestored in an information storage unit included in image processingdevice 3 or an information storage device connected to image processingdevice 3.

Display input device 4 is a so-called touch panel display which combinesa display panel (display device) and a touch panel (position inputdevice). Display input device 4 displays a screen of the composite imageoutput from image processing device 3 and the user performs a touchoperation on the screen so that it is possible to variously adjust eachof camera image regions 12 in the composite image.

Here, in camera unit 2, when assembling an optical system constitutingcamera 1 or when assembling camera 1 itself as a base, an error occursin a mechanical disposition of positions, angles, and the like. Thiserror of the mechanical disposition causes a state in which each ofcamera image regions 12 is not appropriately consistent in the compositeimage obtained by an image composition process and causes deteriorationof a quality of the composite image.

In the present embodiment, by changing a processing condition of theimage process performed by image processing device 3 as calibration forsecuring consistency between camera image regions 12 in the compositeimage, the user adjusts a position, an angle, and a size of camera imageregion 12 so that stitching portions between adjacent camera imageregions 12 in composite image 11 are approximately matched.

In addition, in composite image 11, in some cases, tone and brightnessmay be different at portions where the tones and the brightness shouldbe originally matched in different camera image regions 12. Here, in thepresent embodiment, by changing an imaging condition of camera 1 ascalibration for securing consistency between camera image regions 12 inthe composite image, the user adjusts the tone and the brightness ofcamera image regions 12.

In addition, adjustment work for calibration of ensuring the consistencybetween camera image regions 12 in the composite image is performed bythe touch operation of display input device 4 and the calibration isexecuted by performing a predetermined touch operation on camera imageregion 12 which is a defect.

Next, a schematic configuration of image processing device 3 will bedescribed. FIG. 3 is a functional block diagram illustrating theschematic configuration of image processing device 3. FIG. 4 is anexplanatory diagram illustrating an outline of the stitching processperformed by image processing device 3.

Camera 1 includes lens unit 21, image sensor unit 22, signal processingunit 23, and control unit 24.

Image sensor unit 22 images a subject via lens unit 21 and outputs animage signal. Signal processing unit 23 executes a necessary signalprocess on the image signal output from image sensor unit 22 and outputsa captured image. Control unit 24 controls operations of image sensorunit 22 and signal processing unit 23.

Image processing device 3 includes stitching processing unit (imageprocessing unit) 31, screen generation unit 32, touch operationdetermination unit 33, processing condition setting unit 34, and imageadjustment unit 35. Stitching processing unit 31 includes disparitycorrection amount calculator 41, panoramic image generation unit 42,preview output switching unit 43, disparity correction unit 44, imagecomposition unit 45, and just fit detection unit 46.

Here, only a portion of processing the captured images of two cameras 1is described. As illustrated in FIG. 1, in the present embodiment, theseven cameras 1 are provided. Disparity correction amount calculator 41is provided for each of combinations of the two adjacent cameras 1 andpanoramic image generation unit 42, preview output switching unit 43,and disparity correction unit 44 are provided for each of cameras 1.

Disparity correction amount calculator 41 calculates the disparitycorrection amount which defines a deformation degree of the image duringdisparity correction for each of the frames. Specifically, asillustrated in FIG. 4, each of processes of collimation (projection tocylinder), processing region clipping, and disparity calculation isperformed. In the disparity calculation, disparity (deviation amount) iscalculated by block matching between the two captured images. That is,while shifting the two captured images little by little, a differencebetween the two captured images is calculated and the disparity iscalculated from a positional relationship at which the difference is thesmallest.

Panoramic image generation unit 42 makes the captured imagesrespectively output from cameras 1 to be panoramic images (projection tosphere) and generates the panoramic image (composition source image).First, panoramic image generation unit 42 generates the panoramic imagebased on the preset control parameter (processing condition). Whenprocessing condition setting unit 34 sets a temporary control parameteraccording to the touch operation of the user, panoramic image generationunit 42 generates the panoramic image based on the temporary controlparameter.

Preview output switching unit 43 normally outputs the panoramic imagegenerated by panoramic image generation unit 42 to disparity correctionunit 44 and outputs the panoramic image generated by panoramic imagegeneration unit 42 to image composition unit 45 when displaying apreview display screen. Accordingly, when displaying the preview displayscreen, the image composition process is performed without a disparitycorrection process.

Disparity correction unit 44 performs the disparity correction on thepanoramic image generated by panoramic image generation unit 42 based onthe disparity correction amount output from disparity correction amountcalculator 41 to generate a disparity correction image.

Image composition unit 45 normally performs the image compositionprocess on a plurality of disparity correction images generated bydisparity correction unit 44 to generate the composite image. Inaddition, when displaying the preview display screen, image compositionunit 45 performs the image composition process on a plurality ofpanoramic images generated by panoramic image generation unit 42 togenerate the composite image for preview. The composite image forpreview is normally lower in resolution than the composite image.

When displaying the preview display screen, since only camera imageregion 12 to be adjusted is changed according to the touch operation, itis possible to generate a new composite image by superimposing cameraimage region 12 to be adjusted on the original composite image.

Just fit detection unit 46 detects a just fit state in which consistencyof the stitching portion between camera image region 12 to be adjustedand adjacent camera image region 12 in the composite image is high.

Here, in order to detect the just fit state, an absolute value of adifference between pixel values is obtained between pixels positioned atthe stitching portion of adjacent camera image regions 12, that is, thepixels adjacent to each other across a boundary line of camera imageregion 12, the absolute values of the differences are added, and a sumof the absolute values of the differences are calculated. Then, based ona magnitude of the sum of the absolute values of the differences, adegree (error) of positional deviation of adjacent camera image regions12 is determined and when the sum of the absolute values of thedifferences becomes a minimum value in the vicinity, the just fit stateis determined.

When just fit detection unit 46 detects the just fit state in this way,the composite images at those timings are continuously output within apredetermined range. Accordingly, on the preview display screen, cameraimage region 12 to be adjusted is displayed in a stationary state withinthe predetermined range without interlocking with the touch operation.

Screen generation unit 32 normally generates a regular display screen onwhich the composite image generated by image composition unit 45performing the image composition process on the disparity correctionimage is displayed and outputs the regular display screen to displayinput device 4. In addition, screen generation unit 32 generates thepreview display screen on which the composite image for previewgenerated by image composition unit 45 performing the image compositionprocess on the panoramic image is displayed and outputs the previewdisplay screen to display input device 4.

Touch operation determination unit 33 obtains a detection result of thetouch operation which the user performs on the preview display screendisplayed on display input device 4, from display input device 4. Basedon the detection result of the touch operation, touch operationdetermination unit 33 determines camera image region 12 to be adjustedin composite image 11 and determines an adjustment item (pan adjustment,tilt adjustment, rolling adjustment, and zoom adjustment) according toan operation mode of the touch operation.

In the present embodiment, in a case where the touch operation isperformed with one or two fingers, one camera image region 12 on whichthe touch operation is performed is set as an adjustment target and anindividual adjustment of individually adjusting each of camera imageregions 12 is performed. In addition, in a case where the touchoperation is performed with three or more fingers, all of camera imageregions 12 are set as the adjustment targets and an overall adjustmentof integrally adjusting all of camera image regions 12 is performed.

Further, in a case where the operation mode of the touch operation is atouch operation for different camera image regions 12 with two fingers,two camera image regions 12 on which the touch operation is performedare set as the adjustment targets and it is determined that an imageadjustment is performed between two camera image regions 12.

Based on a determination result of touch operation determination unit33, processing condition setting unit 34 sets a temporary processingcondition related to camera image region 12 to be adjusted. In thepresent embodiment, as the processing condition, the control parameterrelated to generation of the panoramic image (composition source image)by panoramic image generation unit 42 is set.

When processing condition setting unit 34 sets the temporary processingcondition, based on the temporary processing condition, stitchingprocessing unit 31 generates the composition source image from thecaptured image of camera 1 corresponding to camera image region 12 to beadjusted and updates the composite image for preview by performing theimage composition process on the composition source image.

Image adjustment unit 35 performs the image adjustment between twocamera image regions 12. In the present embodiment, the image adjustmentis performed so that at least either tones or brightness in touchregions in two camera image regions 12 is matched. The image adjustmentis executed in a case where the operation mode of the touch operation isa touch operation for different camera image regions 12 with twofingers.

In addition, in the present embodiment, the imaging condition of camera1, that is, the control parameter related to tones and brightness is setso that the tones and the brightness in two camera image regions 12 arematched. The control parameter is output from image adjustment unit 35to control unit 24 of camera 1 and control unit 24 controls image sensorunit 22 and signal processing unit 23 based on the control parameter.

Image processing device 3 illustrated in FIG. 3 is configured by ageneral-purpose information processing device. Each of units of imageprocessing device 3 can be realized by causing a processor to execute aprogram for the image process stored in a memory. In addition, at leasta part of image processing device 3 also can be realized by an exclusivehardware (screen processing circuit) for screen processing.

Next, the disparity correction performed by disparity correction unit 44will be described. FIG. 5 is an explanatory diagram schematicallyillustrating states of an image in a case where the disparity correctionis not executed and a case where the disparity correction is executed.Here, for convenience of explanation, an example of processing theimages of two adjacent cameras 1 will be described.

(A) of FIG. 5 illustrates imaging status by two adjacent cameras 1. Inan example illustrated in (A) of FIG. 5, a person and a mountain whichis a background are simultaneously imaged by two cameras 1.

(B-1) and (B-2) in FIG. 5 illustrate images captured by two cameras 1.As illustrated in (B-1) and (B-2) of FIG. 5, in the captured image bytwo cameras 1, the image of distant view representing the mountain andthe image of close-range view representing the person appear in imageregions of boundary portions corresponding to an overlapping portion ofthe imaging areas of two cameras 1. Here, the mountain of distant viewand the person of close-range view have different distances from camera1, so that in the two captured images, a positional relationship betweenthe image of distant view and the image of close-range view deviates.

(C-1) and (C-2) in FIG. 5 illustrate composite images obtained byperforming simple composite on the images captured by two cameras 1based on the image of distant view. As described above, since thepositional relationship between the image of distant view and the imageof close-range view deviates in the two captured images, when the twocaptured images are simply composited based on the image of distantview, as illustrated in (C-1) and (C-2) of FIG. 5, a defect ofduplicately appearing the image of close-range view or of losing a partof the image of close-range view occurs in the composite image.

Here, a deviation of the image of close-range view when setting theimage of distant view as a standard means a disparity between thecaptured images of two cameras 1. The positional relationship betweenthe images of distant view respectively captured by two cameras 1 can beobtained in advance in a state in which the image of close-range viewdoes not exist and the images are composited based on the information.On the other hand, when the image of close-range view does not exist,the disparity does not exist in the captured images of two cameras 1 andwhen the image of close-range view exists, the disparity between thecaptured images of two cameras 1 exists.

Therefore, in a state in which the image of distant view and the imageof close-range view appear in the image regions of the boundary portionsin the captured images of two cameras 1, the disparity generated betweenthe two captured images causes a defect like the image of close-rangeview duplicately appears in the composite image. Then, the disparitycorrection for removing the defect is necessary. There arerepresentatively two types of correction methods for the disparitycorrection. (C-3) in FIG. 5 illustrates a first correction method and(C-4) in FIG. 5 illustrates a second correction method.

As illustrated in (C-3) of FIG. 5, in the first correction method, theimage is deformed so that the image region in which the image ofclose-range view mainly appears is horizontally shifted, and then, thecaptured images of two cameras 1 are composited so as to consist thepositional relationship of the image of distant view and the image ofclose-range view between the captured images of two cameras 1.

As illustrated in (C-4) of FIG. 5, in the second correction method, acurved stitching boundary is set so as to avoid the image region inwhich the image of close-range view appears. After trimming is performedto clip the captured images of two cameras 1 along the stitchingboundary, the images are composited.

In this way, by performing the disparity correction, it is possible togenerate the appropriate image. In the present embodiment, the firstcorrection method, that is, the disparity correction of deforming theimage to horizontally shift the image region in which the image ofclose-range view mainly appears is performed on the captured images oftwo cameras 1 so that the positional relationship of the image ofdistant view and the image of close-range view between the capturedimages of two cameras 1 consists.

Next, an art of adjusting camera image regions 12 will be described.FIG. 6 is an explanatory diagram illustrating the art of adjustingcamera image regions 12.

In the present embodiment, when displaying the preview display screen,panoramic image generation unit 42 makes the captured images obtainedfrom cameras 1 to be the panoramic image (projection to sphere) andgenerates the panoramic image (composition source image). Then, theimages are composited by clipping a necessary region from the panoramicimage. At this time, a position and a size of each of camera imageregions 12 on composite image 11 are defined according to a projectionposition (central position of projection region) and a projection size(size of projection region) during making the panoramic image.

On the other hand, by each of cameras 1 imaging the subject with a wideangle, in captured image 51, blank region 53 is formed outside usedregion 52 corresponding to camera image region 12 finally displayed onthe composite image. It is possible to adjust the projection positionand the projection size during making the panoramic image within a rangeof blank region 53.

Here, in the present embodiment, it is possible to perform the panadjustment, tilt adjustment, rolling adjustment, and zoom adjustment bythe touch operation.

As illustrated in (A) of FIG. 6, in the pan adjustment (positionadjustment in horizontal direction), camera image region 12 can be movedin the horizontal direction and at this time, a process of adjusting theprojection position of the image in the horizontal direction isperformed. As illustrated in (B) of FIG. 6, in the tilt adjustment(position adjustment in vertical direction), camera image region 12 canbe moved in the vertical direction and at this time, a process ofadjusting the projection position of the image in the vertical directionis performed. As illustrated in (C) of FIG. 6, in the rolling adjustment(position adjustment in rotation direction), camera image region 12 canbe rotated and at this time, a process of adjusting the projectionposition of the image in the rotation direction is performed. Asillustrated in (D) of FIG. 6, in the zoom adjustment(extension/contraction adjustment), camera image region 12 can be zoomedout or zoomed in and at this time, a process of adjusting the size ofthe image is performed.

In addition, in the present embodiment, processing condition settingunit 34 sets a setting value related to the projection position and theprojection size as the control parameter related to generation of thepanoramic image (composition source image) according to the touchoperation. Based on the control parameter, panoramic image generationunit 42 adjusts the projection position and the projection size duringmaking the panoramic image.

Here, regarding each of camera image regions 12, the control parameterrelated to each of the adjustment items of the pan adjustment, the tiltadjustment, the rolling adjustment, and the zoom adjustment is preset.When the touch operation related to one of the pan adjustment, the tiltadjustment, the rolling adjustment, and the zoom adjustment isperformed, the control parameter related to the adjustment itemcorresponding to camera image region 12 to be adjusted is updated.

Next, the adjustment item according to the operation mode of the touchoperation will be described. FIG. 7 is an explanatory diagramillustrating a list of the adjustment items according to the operationmode of the touch operation.

In the present embodiment, in a case where the touch operation isperformed with one or two fingers, one camera image region 12 on whichthe touch operation is performed is set as an adjustment target and anindividual adjustment of individually adjusting each of camera imageregions 12 can be performed. In addition, in a case where the touchoperation is performed with three or more fingers, all of camera imageregions 12 are set as the adjustment targets and an overall adjustmentof integrally adjusting all of camera image regions 12 can be performed.

In addition, in the present embodiment, the pan adjustment (positionadjustment in horizontal direction) of horizontally moving camera imageregion 12 to be adjusted, the tilt adjustment (position adjustment invertical direction) of vertically moving camera image region 12 to beadjusted, the rolling adjustment (position adjustment in rotationdirection) of rotating camera image region 12 to be adjusted, and thezoom adjustment (extension/contraction adjustment) of zooming in orzooming out camera image region 12 to be adjusted can be performed.

Hereinafter, the adjustment item according to the operation mode of thetouch operation will be described in detail.

First, a case where each of camera image regions 12 is individuallyadjusted will be described. FIG. 8 is an explanatory diagramillustrating the touch operation in a case of individually adjustingeach of camera image regions 12 of horizontal cameras 1. FIG. 9 is anexplanatory diagram illustrating the touch operation in a case ofindividually adjusting camera image regions 12 of upward cameras 1.

In the present embodiment, in a case where the touch operation isperformed with one or two fingers, one camera image region 12 on whichthe touch operation is performed is set as an adjustment target and itis possible to individually adjust each of camera image regions 12.

First, in a case of camera image region 12 related to horizontal camera1, as illustrated in (A) of FIG. 8, it is possible to perform the panadjustment (position adjustment in horizontal direction) of horizontallymoving camera image region 12 to be operated by a swipe operation ofsliding one finger touched on the screen in the horizontal direction(left and right direction of screen).

In addition, as illustrated in (B) of FIG. 8, it is possible to performthe tilt adjustment (position adjustment in vertical direction) ofvertically moving camera image region 12 to be operated by a swipeoperation of sliding one finger touched on the screen in the verticaldirection (up and down direction of screen).

In addition, as illustrated in (C) of FIG. 8, it is possible to performthe rolling adjustment (position adjustment in rotation direction) ofrotating camera image region 12 to be operated by a rotation operationof moving two fingers touched on the screen to draw a circle.

In addition, as illustrated in (D) of FIG. 8, it is possible to performthe zoom adjustment (extension/contraction adjustment) of zooming in orzooming out camera image region 12 to be operated by a pinch operationof moving two fingers touched on the screen to widen or narrow a gapbetween the two fingers.

On the other hand, in a case of camera image region 12 related to upwardcamera 1, as illustrated in (A-1) of FIG. 9, it is possible to move theimage which appears in camera image region 12 in the horizontaldirection by a swipe operation of sliding one finger touched on thescreen in the horizontal direction (left and right direction of image).

Here, in camera image region 12 related to upward camera 1, a horizontalmovement on composite image 11 corresponds to rotating of projectionregion 62 in a latitude line direction on projection sphere 61 asillustrated in (A-2) of FIG. 9. For this reason, when the horizontalswipe operation is performed, the rolling adjustment (positionadjustment in rotation direction) of adjusting the projection positionof the image in the rotation direction is performed.

In addition, as illustrated in (B-1) of FIG. 9, it is possible to movethe image which appears in camera image region 12 in the verticaldirection by a swipe operation of sliding one finger touched on thescreen in the vertical direction (up and down direction of image).

Here, in camera image region 12 related to upward camera 1, a verticalmovement on composite image 11 corresponds to moving of projectionregion 62 in a longitude line direction on projection sphere 61 asillustrated in (B-2) of FIG. 9. For this reason, when the vertical swipeoperation is performed, the tilt adjustment (position adjustment indiameter direction) of adjusting the projection position of the image inthe vertical direction is performed.

Next, a case of integrally adjusting all of camera image regions 12 willbe described. FIGS. 10 and 11 are explanatory diagrams illustrating thetouch operation in the case of integrally adjusting all of camera imageregions 12.

In the present embodiment, in a case where the touch operation isperformed with three or more fingers, all of camera image regions 12 areset as the adjustment targets and it is possible to integrally adjustall of camera image regions 12.

First, as illustrated in (A-1) of FIG. 10, it is possible to perform thepan adjustment (position adjustment in horizontal direction) ofhorizontally moving all of camera image regions 12 by a swipe operationof sliding three or more fingers touched on the screen in the horizontaldirection (left and right direction of screen). At this time, accordingto the movement of camera image region 12 related to horizontal camera1, the image which appears in camera image region 12 related to upwardcamera 1 moves in the horizontal direction.

In this case, by performing the swipe operation greatly, camera imageregion 12 related to horizontal camera 1 is displayed so as to becirculated.

Accordingly, for example, in the state illustrated in (A-1) of FIG. 10,camera image region 12 positioned at a left or right end can be arrangedat a center of the screen as illustrated in (A-2) of FIG. 10.

In addition, as illustrated in (B-1) of FIG. 10, it is possible toperform the tilt adjustment (position adjustment in vertical direction)of vertically moving all of camera image regions 12 by a swipe operationof sliding three or more fingers touched on the screen in the verticaldirection (up and down direction of screen).

In this case, by performing the swipe operation greatly, camera imageregion 12 is displayed so as to be vertically circulated. Accordingly,for example, in the state illustrated in (B-1) of FIG. 10, camera imageregion 12 related to upward camera 1 positioned at an upper end can bearranged at a center of the screen as illustrated in (B-2) of FIG. 10.In this case, by sandwiching camera image region 12 related to upwardcamera 1, camera image region 12 related to upward camera 1 is displayedin a state in which camera image region 12 is divided into two.

In addition, as illustrated in (A) of FIG. 11, it is possible to performthe rolling adjustment (position adjustment in rotation direction) ofrotating all of camera image regions 12 by a rotation operation ofmoving three or more fingers touched on the screen to draw a circle.

In addition, as illustrated in (B) of FIG. 11, it is possible to performthe zoom adjustment (extension/contraction adjustment) of zooming in orzooming out all of camera image regions 12 by the pinch operation ofmoving three or more fingers touched on the screen to widen or narrow agap between three or more fingers.

Next, a flow of a process executed by image processing device 3according to the touch operation of the user will be described. FIGS.12, 13, and 14 are flowcharts illustrating the flow of the processexecuted by image processing device 3 according to the touch operationof the user.

First, touch operation determination unit 33 determines whether or notthe touch operation is detected (ST 101). Here, in a case where thetouch operation is detected (Yes in ST 101), the disparity correction isreleased (ST 102). Then, it is determined whether or not the touchoperation is an operation with one finger (ST 103).

Here, in a case where the touch operation is the operation with onefinger (Yes in ST 103), next, it is determined whether or not theoperation target is camera image region 12 related to upward camera 1(ST 104).

Here, in a case where the operation target is not camera image region 12related to upward camera 1, that is, the operation target is cameraimage region 12 related to horizontal camera 1 (No in ST 104), next, itis determined whether or not the touch operation is the swipe operationin the horizontal direction (ST 105). Here, in the case of the swipeoperation in the horizontal direction (Yes in ST 105), processingcondition setting unit 34 executes the pan adjustment (positionadjustment in horizontal direction) according to a finger movement andupdates the control parameter of pan (ST 106).

Here, in a case where the touch operation is not the swipe operation inthe horizontal direction (No in ST 105), next, it is determined whetheror not the touch operation is the swipe operation in the verticaldirection (ST 107). Here, in the case of the swipe operation in thevertical direction (Yes in ST 107), processing condition setting unit 34executes the tilt adjustment (position adjustment in vertical direction)according to the finger movement and updates the control parameter oftilt (ST 108).

On the other hand, in a case where the operation target is camera imageregion 12 related to upward camera 1 (Yes in ST 104), next, it isdetermined whether or not the touch operation is the swipe operation inthe horizontal direction (ST 109). Here, in the case of the swipeoperation in the horizontal direction (Yes in ST 109), processingcondition setting unit 34 executes the rolling adjustment (positionadjustment in rotation direction) according to the finger movement andupdates the control parameter of rolling (ST 110).

Here, in a case where the touch operation is not the swipe operation inthe horizontal direction (No in ST 109), next, it is determined whetheror not the touch operation is the swipe operation in the verticaldirection (ST 111). Here, in the case of the swipe operation in thevertical direction (Yes in ST 111), processing condition setting unit 34executes the tilt adjustment (position adjustment in vertical direction)according to the finger movement and updates the control parameter oftilt (ST 112).

On the other hand, in a case where the touch operation is not theoperation with one finger (No in ST 103), next, it is determined whetheror not the touch operation is the operation with two fingers (ST 113).

Here, in a case where the touch operation is the operation with twofingers (Yes in ST 113), next, it is determined whether or not the touchoperation is the rotation operation (ST 114). Here, in the case of therotation operation (Yes in ST 114), processing condition setting unit 34executes the rolling adjustment (position adjustment in rotationdirection) according to the finger movement and updates the controlparameter of rolling (ST 115).

In addition, in a case where the touch operation is not the rotationoperation (No in ST 114), next, it is determined whether or not thetouch operation is the pinch operation (ST 116). Here, in the case ofthe pinch operation (Yes in ST 116), processing condition setting unit34 executes the zoom adjustment (extension/contraction adjustment)according to the finger movement and updates the control parameter ofzoom (ST 117).

On the other hand, in a case where the touch operation is not theoperation with two fingers (No in ST 113), it is determined that thetouch operation is the operation with three or more fingers and next, itis determined whether or not the touch operation is the swipe operationin the horizontal direction (ST 118). Here, in the case of the swipeoperation in the horizontal direction (Yes in ST 118), processingcondition setting unit 34 executes the pan adjustment (positionadjustment in horizontal direction) according to a finger movement andupdates the control parameter of pan (ST 119).

Here, in a case where the touch operation is not the swipe operation inthe horizontal direction (No in ST 118), next, it is determined whetheror not the touch operation is the swipe operation in the verticaldirection (ST 120). Here, in the case of the swipe operation in thevertical direction (Yes in ST 120), processing condition setting unit 34executes the tilt adjustment (position adjustment in vertical direction)according to the finger movement and updates the control parameter oftilt (ST 121).

On the other hand, in a case where the touch operation is not the swipeoperation in the vertical direction (No in ST 120), next, it isdetermined whether or not the touch operation is the rotation operation(ST 122). Here, in the case of the rotation operation (Yes in ST 122),processing condition setting unit 34 executes the rolling adjustment(position adjustment in rotation direction) according to the fingermovement and updates the control parameter of rolling (ST 123).

In addition, in a case where the touch operation is not the rotationoperation (No in ST 122), next, it is determined whether or not thetouch operation is the pinch operation (ST 124). Here, in the case ofthe pinch operation (Yes in ST 124), processing condition setting unit34 executes the zoom adjustment (extension/contraction adjustment)according to the finger movement and updates the control parameter ofzoom (ST 125).

As described above, when the control parameters of the pan, the tilt,the rolling, and the zoom are updated by the touch operation of theuser, panoramic image generation unit 42 of stitching processing unit 31performs the process of generating the panoramic image based on theupdated control parameter and next, image composition unit 45 performsthe process of compositing the panoramic images and the generatedcomposite image is displayed on the preview display screen (ST 126).

Next, an image process of improving visibility of camera image region 12to be adjusted will be described. FIG. 15 is an explanatory diagramillustrating the image process of improving the visibility of cameraimage region 12 to be adjusted.

In the present embodiment, composite image 11 changes a tone of cameraimage region 12 not to be adjusted from the initial tone, so that theimage process is performed on image composition unit 45 to excellentlyimprove visibility of camera image region 12 to be adjusted.Specifically, in the present embodiment, at least an image process ofchanging luminance is performed on camera image region 12 not to beadjusted. In the example illustrated in FIG. 15, the image process oflowering the luminance is performed on camera image region 12 not to beadjusted and camera image region 12 not to be adjusted is grayed out anddarkly displayed.

Accordingly, in the preview display screen on which composite image 11is displayed, camera image region 12 to be adjusted becomes conspicuousand the user can intuitively grasp camera image region 12 to beadjusted.

Next, control of causing the user to recognize the just fit state ofcamera image region 12 to be adjusted will be described. FIG. 16 is anexplanatory diagram illustrating the control of causing the user torecognize the just fit state of camera image region 12 to be adjusted.

In the present embodiment, when a touch operation of adjusting cameraimage region 12 is performed on the preview display screen, theprocessing condition (control parameter related to generation ofpanoramic image) of stitching processing unit 31 is regularly updatedaccording to the touch operation, the composition source image isupdated according to this, and the composite image composited from thecomposition source images is regularly output from stitching processingunit 31. Accordingly, the composite image is displayed on the previewdisplay screen as a video, at this time, an animation in which cameraimage region 12 to be operated changes in conjunction with the fingermovement is executed.

In addition, in the present embodiment, just fit detection unit 46detects the just fit state in which consistency of the stitching portionbetween camera image region 12 to be operated and adjacent camera imageregion 12 is high. Then, when just fit detection unit 46 detects thejust fit state, the animation of camera image region 12 to be operated,that is, the movement of camera image region 12 in conjunction with thefinger movement is stopped.

At this time, image composition unit 45 continuously outputs thecomposite image within a predetermined range based on the compositionsource image (panoramic image) obtained at a timing at which the justfit state is detected. Accordingly, on the preview display screen,camera image region 12 to be operated is displayed in a stationary statewithout interlocking with the finger movement.

In addition, the animation is stopped within a predetermined range andin a case of an outside of the predetermined range, the stop ofanimation is released. In this case, a timing of releasing the stop ofthe animation can be defined by a distance. That is, when the distancefrom a position of the just fit state to a touch position becomes largerthan a predetermined distance, the stop of the animation is released. Inaddition, the timing of releasing the stop of the animation can bedefined by a time. That is, when a predetermined time (for example, 1second) elapses after the just fit state is detected, the stop of theanimation is released.

In the example illustrated in FIG. 16, the swipe operation of slidingone finger touched on the screen in the vertical direction (up and downdirection of image) is performed. As illustrated in (A) of FIG. 16, whenthe finger moves, camera image region 12 to be operated is moved in thevertical direction according to the finger movement. Then, at the timingillustrated in (B) of FIG. 16, when the just fit state is detected, asillustrated in (C) of FIG. 16, even if the finger further moves, cameraimage region 12 to be operated is stopped as it is. As illustrated in(D) of FIG. 16, when the finger further moves, camera image region 12 tobe operated is moved in the vertical direction according to the fingermovement.

In the present embodiment, when the just fit state of camera imageregion 12 to be adjusted is detected, the animation of camera imageregion 12 is stopped. Accordingly, the user can intuitively grasp thejust fit state and it is possible to quickly perform the adjustmentoperation.

In the present embodiment, by stopping the animation of camera imageregion 12 to be operated, the user recognizes the just fit state. Amethod of causing the user to recognize the just fit state is notlimited thereto. For example, at the timing of the just fit, cameraimage region 12 may be flashed (blinked). In addition, at the timing ofthe just fit, a frame line surrounding camera image region 12 may bedisplayed.

Next, a touch operation when a tone and brightness of camera imageregion 12 are adjusted will be described. FIG. 17 is an explanatorydiagram illustrating the touch operation when the tone and thebrightness of camera image region 12 are adjusted.

In the present embodiment, in a case where the touch operation issimultaneously performed on two camera image regions 12 with twofingers, two camera image regions 12 on which the touch operation isperformed are set as the adjustment targets and image adjustment unit 35performs the image adjustment between two camera image regions 12.

Specifically, in the present embodiment, the image adjustment isperformed so that the tones and the brightness of touch regions 71(predetermined range around touch position) in two camera image regions12 on which the touch operation is performed are matched. In the exampleillustrated in FIG. 17, the touch operation is performed in a region ofa sky in two camera image regions 12 so that the tones and thebrightness of the sky in two camera image regions 12 are matched.

Accordingly, in a case where a difference in the tones and thebrightness occurs in two camera image regions 12 in the region in whichthe tones and the brightness should be originally matched, by performingthe touch operation on the region, it is possible to easily match thetones and the brightness of two camera image regions 12.

In the image adjustment, information on the tones and brightness of thetouch region in two camera image regions 12 to be adjusted is obtainedand camera 1 of an imaging source of two camera image regions 12 iscontrolled based on the information so that the tones and the brightnessof the touch region in two camera image regions 12 are matched.

At this time, the imaging condition of camera 1, that is, the controlparameter related to tones and brightness is adjusted so that the tonesand the brightness in two camera image regions 12 are matched.Specifically, a white balance gain value in signal processing unit 23 isadjusted as a control parameter related to the tone (white balance). Inaddition, a shutter value (shutter speed) and a sensor gain value inimage sensor unit 22, and a digital gain value in signal processing unit23 are adjusted as control parameters related to brightness (exposure).

In the present embodiment, the touch operation is simultaneouslyperformed on two camera image regions 12 with two fingers, so that theimage adjustment of two camera image regions 12 is performed. When thetouch operation is simultaneously performed on three or more cameraimage regions 12 with three or more fingers, the image adjustment ofthree or more camera image regions 12 can be performed. In addition, inthe present embodiment, the control is performed so that both of thetones and the brightness are matched, but one of the tones and thebrightness, for example, the control may be performed so that only thetones are matched.

As described above, the embodiment is described as an example of atechnique disclosed in the present application. However, the techniquein the present disclosure is not limited thereto. The technique also canbe applied to the embodiment being changed, exchanged, added, andomitted. In addition, it is also possible to form a new embodiment bycombining components in the embodiment described above.

For example, in the embodiment described above, the image processingdevice (including PC) provided separately from the camera unit havingthe plurality of cameras performs the image composition process or thelike and outputs the composite image, but the embodiment can beconfigured as the camera unit (imaging device) mounting the imageprocessing device.

In addition, the embodiment described above is configured to include theupward camera and the horizontal camera, but the embodiment may beconfigured to include a downward camera instead of the upward camera.Further, the embodiment may be configured to include both of the upwardcamera and the downward camera.

In addition, in the embodiment described above, when the preview displayscreen is displayed, a preview output switching unit outputs the inputcomposition source image (panoramic image) to the image composition unitand the composition source image is not input to the disparitycorrection unit, but after the composition source image is input to thedisparity correction unit, by setting the disparity correction amount to0, the disparity correction process may not be actually performed.

Further, in the embodiment described above, the image processing devicewhich generates the normal composite image executes the calibration forsecuring consistency between the camera image regions in the compositeimage, but the embodiment may be configured to include an exclusivedevice for the calibration, an exclusive program for the calibration maybe installed in the information processing device, and the calibrationmay be executed.

INDUSTRIAL APPLICABILITY

An image processing device, an imaging system that includes the imageprocessing device, and a calibration method according to the presentdisclosure are advantageous in that a user can conveniently performadjustment work for calibration of securing consistency between cameraimage regions in a composite image, and are useful as an imageprocessing device that outputs a composite image by performing an imagecomposition process on an image for each of a plurality of cameras, animaging system that includes the image processing device, a calibrationmethod of securing consistency between camera image regions in thecomposite image, and the like.

REFERENCE MARKS IN THE DRAWINGS

-   1 CAMERA-   2 CAMERA UNIT-   3 IMAGE PROCESSING DEVICE-   4 DISPLAY INPUT DEVICE-   11 COMPOSITE IMAGE-   12 CAMERA IMAGE REGION-   31 STITCHING PROCESSING UNIT (IMAGE PROCESSING UNIT)-   32 SCREEN GENERATION UNIT-   33 TOUCH OPERATION DETERMINATION UNIT-   34 PROCESSING CONDITION SETTING UNIT-   35 IMAGE ADJUSTMENT UNIT-   41 DISPARITY CORRECTION AMOUNT CALCULATOR-   42 PANORAMIC IMAGE GENERATION UNIT-   43 PREVIEW OUTPUT SWITCHING UNIT-   44 DISPARITY CORRECTION UNIT-   45 IMAGE COMPOSITION UNIT-   46 JUST FIT DETECTION UNIT

What is claimed is:
 1. An image processing device that outputs acomposite image by performing an image composition process on an imagefor each of a plurality of cameras, the image processing devicecomprising: a processor; and a memory including instructions that, whenexecuted by the processor, cause the processor to perform operations,the operations including: generating a plurality of composition sourceimages from the image for each of the plurality of cameras based on apreset processing condition; performing the image composition process onthe plurality of composition source images to generate a composite imagefor preview, the composite image including a plurality of camera imageregions, each of the plurality of camera image regions corresponding toone of the plurality of composition source images and one of theplurality of cameras; generating a preview display screen on which thecomposite image is displayed and outputting the preview display screento a display; detecting a first touch operation that a user performs onthe preview display screen displayed on the display and determining afirst adjustment item according to the first touch operation, the firsttouch operation being performed with one or two fingers and in relationto a first camera image region of the plurality of camera image regions,the first camera image region corresponding to a first compositionsource image of the plurality of composition source images and a firstcamera of the plurality of cameras; first adjusting at least one of thefirst composition source image or the first camera according to thefirst adjustment item and updating the composite image; detecting asecond touch operation that the user performs on the preview displayscreen displayed on the display and determining a second adjustmentitem, the second touch operation being performed with at least threefingers; and second adjusting at least one of all composition sourceimages of the plurality of composition source images or all cameras ofthe plurality of cameras according to the second adjustment item andupdating the composite image.
 2. The image processing device of claim 1,wherein the operations further include: detecting a third touchoperation that the user performs on the preview display screen displayedon the display and determining a third adjustment item, the third touchoperation being performed with two fingers and in relation to two cameraimage regions; and in a case where the third touch operation isperformed, performing an image adjustment between the two camera imageregions according to the third adjustment item.
 3. The image processingdevice of claim 1, wherein the operations further include: in a casewhere the first touch operation is performed, setting the first cameraimage region on which the first touch operation is performed as anadjustment target and determining the first adjustment item according tothe first touch operation.
 4. The image processing device of claim 3,wherein the first adjusting includes: performing a pan adjustment in acase where the first touch operation is a swipe operation in ahorizontal direction with one finger, performing a tilt adjustment in acase where the first touch operation is a swipe operation in a verticaldirection with one finger, performing a rolling adjustment in a casewhere the first touch operation is a rotation operation with twofingers, and performing a zoom adjustment in a case where the firsttouch operation is a pinch operation with two fingers.
 5. The imageprocessing device of claim 1, wherein the operations further include:performing image processing of changing a tone of at least one of theplurality of camera image regions not to be adjusted in the compositeimage from an initial tone.
 6. The image processing device of claim 5,wherein the operations further include: performing image processing ofchanging at least luminance on the at least one of the plurality ofcamera image regions not to be adjusted.
 7. The image processing deviceof claim 1, wherein the operations further include: in a case where thesecond touch operation is performed with at least three fingers, settingall of the plurality of camera image regions on which the second touchoperation is performed as adjustment targets and determining the secondadjustment item according to the second touch operation.
 8. The imageprocessing device of claim 1, wherein the second adjusting includes:performing a pan adjustment in a case where the second touch operationis a swipe operation in a horizontal direction with at least threefingers, performing a tilt adjustment in a case where the second touchoperation is a swipe operation in a vertical direction with at leastthree fingers, performing a rolling adjustment in a case where thesecond touch operation is a rotation operation with at least threefingers, performing a zoom adjustment in a case where the second touchoperation is a pinch operation with at least three fingers.
 9. The imageprocessing device of claim 1, wherein the operations further include:detecting a just fit state in which consistency of a stitching portionbetween the first camera image region to be adjusted and a second cameraimage region adjacent to the first camera image region to be adjusted inthe composite image is high, and in a case of detecting that the justfit state is high, outputting the composite image within a predeterminedrange at a timing at which the just fit state is detected.
 10. The imageprocessing device of claim 1, wherein the operations further include:detecting a third touch operation that the user performs on the previewdisplay screen displayed on the display and determining a thirdadjustment item, the third touch operation being performed with twofingers and in simultaneous relationship to two camera image regions;and in a case where the third touch operation is performed, performingan image adjustment so that at least either tones or brightness in thetwo camera image regions is matched.
 11. The image processing device ofclaim 1, wherein the operations further include: performing a disparitycorrection process on the plurality of composition source images beforethe image composition process; and after the preview display screen isdisplayed, the disparity correction process is not performed on theplurality of composition source images.
 12. An imaging systemcomprising: the image processing device according to claim 1; theplurality of cameras; and the display.
 13. The image processing deviceof claim 1, wherein, in the second adjusting, all cameras of theplurality of cameras are adjusted according to the second adjustmentitem.
 14. A calibration method of securing consistency between cameraimage regions in a composite image generated by performing an imagecomposition process on an image for each of a plurality of cameras, thecalibration method comprising: generating a plurality of compositionsource images from the image for each of the plurality of cameras basedon a preset processing condition and performing the image compositionprocess on the plurality of composition source images to generate acomposite image for preview, the composite image including a pluralityof camera image regions, each of the plurality of camera image regionscorresponding to one of the plurality of composition source images andone of the plurality of cameras; generating a preview display screen onwhich the composite image is displayed and outputting the previewdisplay screen to a display; detecting a first touch operation which auser performs on the preview display screen displayed on the display anddetermining a first adjustment item according to the first touchoperation, the first touch operation being performed with one or twofingers and in relation to a first camera image region of the pluralityof camera image regions, the first camera image region corresponding toa first composition source image of the plurality of composition sourceimages and a first camera of the plurality of cameras; adjusting atleast one of the first composition source image or the first cameraaccording to the first adjustment item and updating the composite image;detecting a second touch operation that the user performs on the previewdisplay screen displayed on the display and determining a secondadjustment item, the second touch operation being performed with atleast three fingers; and adjusting at least one of all compositionsource images of the plurality of composition source images or allcameras of the plurality of cameras according to the second adjustmentitem and updating the composite image.